Systems and methods for data evaluation and classification

ABSTRACT

The present disclosure is directed towards systems and methods for evaluating and classifying data, which comprises receiving a first signal associated with a set of user defined categories and a second signal associated with a first set of one or more data items to be analyzed. The set of categories are presented on an interface and each of the data items are to be analyzed are presented on a tile overlay on the interface. A first swipe identification signal identifying a first category to apply to a first data item and a second swipe identification signal identifying a second category to apply to a second data item are received. Based upon the swipe identification signals, a first record associating the first data item with the first category and a second record associating the second data item with the second category are generated and stored in a memory.

COPYRIGHT NOTICE

A portion of the disclosure of this patent document contains material,which is subject to copyright protection. The copyright owner has noobjection to the facsimile reproduction by anyone of the patent documentor the patent disclosure, as it appears in the Patent and TrademarkOffice patent files or records, but otherwise reserves all copyrightrights whatsoever. The following notice applies to this document:Copyright© 2015 Thomson Reuters.

TECHNICAL FIELD

This disclosure relates generally to the evaluation and classificationof data. More specifically, the disclosure is directed towards systemsand methods for evaluating and classifying data files, such asdocuments, document segments, text, text segments, images image segmentsand the like.

BACKGROUND

Electronic classification and analysis tools are invaluable resourcesthat are used in nearly every business, legal, scientific and academicenvironment to evaluate, review and organize pertinent documents ordocument excerpts. For example, in the legal environment, electronicclassification is a powerful and efficient tool for e-discovery andcontract analysis. Such tools, however, do have limitations, typicallyrequiring a personal computer with peripheral devices, such as thestandard mouse and keyboard in order to interact with such tools and toevaluate and classify data items. Yet, with the recent gain inpopularity of tablet devices and touch displays on computers, more andmore end users typically limit their reliance on or do not utilize suchperipheral devices, relying instead primarily on human gesture, such asfinger swiping.

Accordingly, there exists a need for evaluating and classifying datasets and portions of data sets in an efficient manner without thenecessity of using peripheral devices in conjunction with readily usedand available touch sensitive devices.

SUMMARY

The present disclosure is directed towards systems and methods forevaluating and classifying data sets and portions of data sets. In oneaspect, the method includes receiving a first signal associated with aset of categories, the set of categories being user defined, and asecond signal associated with a first set of one or more data items tobe analyzed. Presented on a graphical user interface are the set ofcategories and each of the one or more data items of the first set to beanalyzed on a tile overlay. A first swipe identification signal is thenreceived, the first swipe identification signal identifying a firstcategory from the set of categories to apply to a first data item fromthe set of one or more data items to be analyzed. A second swipeidentification signal is then received, the second swipe identificationsignal identifying a second category from the set of categories to applyto a second data item from the set of one or more data items to beanalyzed. Based upon the first and second swipe identification signals,a first record and a second record are generated and stored in a memory,the first record associating the first data item with the first categoryand the second record associating the second data item with the secondcategory.

According to one embodiment, the first category and the second categoryare the same. In one embodiment, the method further includes, generatinga spreadsheet file comprising the first and second records. In oneembodiment, the method further includes displaying an administrationinterface tool comprising an UI administration interface, a deckadministration interface and an user administration interface. Accordingto one embodiment, the method further includes populating a categoryname field, a category label field and a screen location field in the UIadministration interface for each category of the set of categories tobe assigned a data item, the screen location field populated with ascreen location value representative of the location on the graphicaluser interface of a visual representation of each category. The methodfurther includes populating an end user name field and an end user emailaddress field for one or more end users in the user administrationinterface and assigning an end user identification, an end user passwordand an end user role to each of the one or more end users in the useradministration interface. In one embodiment, the method further includesgenerating, using the deck administration interface, one or more sets ofone or more data items to be analyzed and assigning, using the deckadministration interface, one or more end users to the one or more setsof one or more data items to be analyzed.

A system, as well as articles that include a machine-readable mediumstoring machine-readable program code for implementing the varioustechniques, are disclosed. Details of various embodiments are discussedin greater detail below.

Additional features and advantages will be readily apparent from thefollowing detailed description, the accompanying drawings and theclaims.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a schematic depicting an exemplary computer-based system forevaluating and classifying data items;

FIG. 2 is a schematic depicting an exemplary computer-based system forevaluating and classifying data items;

FIG. 3 is a flow diagram illustrating an exemplary computer-implementedmethod for evaluating and classifying data items;

FIG. 4 is a screen diagram of an exemplary website portal for a toolused to evaluate and classify data items;

FIG. 5 is a screen diagram of an exemplary UI Administration interfacetool used to evaluate and classify data items;

FIG. 6 is a screen diagram of an exemplary User Administration interfacetool used to evaluate and classify data items;

FIG. 7 is a flow diagram illustrating an exemplary computer-implementedmethod for evaluating and classifying data items;

FIG. 8 is a screen diagram of an exemplary interface tool used toevaluate and classify data items illustrating a pre-classification view;

FIG. 9 is a screen diagram of an exemplary interface tool used toevaluate and classify data items illustrating a post-classificationview;

FIG. 10 is a screen diagram of an exemplary interface tool used toevaluate and classify data items illustrating a post-classificationview;

FIG. 11 is a screen diagram of an exemplary interface tool used toevaluate and classify data items illustrating a post-classificationview;

FIG. 12 is a screen diagram of an exemplary interface tool used toevaluate and classify data items illustrating a post-classificationview;

FIG. 13 is a screen diagram of an exemplary administration interfacetool used to evaluate and classify data items;

FIG. 14 is a screen diagram of an exemplary interface tool used toevaluate and classify data items illustrating a user's assigned tasks;and

FIG. 15 is a screen diagram of an exemplary report outlining exemplaryclassification statistics and results from the exemplary interface tool.

Like reference symbols in the various drawings indicate like elements.

DETAILED DESCRIPTION

In the following description, reference is made to the accompanyingdrawings that form a part hereof, and in which is shown by way ofillustration specific embodiments in which the disclosure may bepracticed. It is to be understood that other embodiments may be utilizedand structural changes may be made without departing from the scope ofthe present disclosure.

Turning now to FIG. 1, an example of a suitable computing system 100within which embodiments of the disclosure may be implemented ispresented. The computing system 100 is only one example and is notintended to suggest any limitation as to the scope of use orfunctionality of the disclosure. Neither should the computing system 100be interpreted as having any dependency or requirement relating to anyone or combination of illustrated components.

For example, the present disclosure is operational with numerous othergeneral purpose or special purpose computing consumer electronics,network PCs, minicomputers, mainframe computers, laptop computers, aswell as distributed computing environments that include any of the abovesystems or devices, and the like.

The disclosure may be described in the general context ofcomputer-executable instructions, such as program modules, beingexecuted by a computer. Generally, program modules include routines,programs, objects, components, data structures, loop code segments andconstructs, and other computer instructions known to those skilled inthe art that perform particular tasks or implement particular abstractdata types. The disclosure can be practiced in distributed computingenvironments where tasks are performed by remote processing devices thatare linked through a communications network. In a distributed computingenvironment, program modules are located in both local and remotecomputer storage media including memory storage devices. Tasks performedby the programs and modules are described below and with the aid offigures. Those skilled in the art may implement the description andfigures as processor executable instructions, which may be written onany form of a computer readable media.

In one embodiment, with reference to FIG. 1, the system 100 includes aserver device 110 configured to include a processor 112, such as acentral processing unit (“CPU”), random access memory (“RAM”) 114, oneor more input-output devices 116, such as a display device (not shown)and keyboard (not shown), non-volatile memory 120 and data store 130,all of which are interconnected via a common bus and controlled by theprocessor 112.

As shown in the FIG. 1 example, in one embodiment, the non-volatilememory 120 is configured to include an administrative module 122 and anevaluation module 124. The administrative module 122 is used to verifythe administrator and allow access to the system 100, as well as togenerate sets of data items (referred to herein as “decks” or “decks ofcards”), assign those decks to end user reviewers and to generatesummary reports and provides finalized categorization assignments todata items. The evaluation module 124 is used to receive swipeidentification signals corresponding to end users' assignment of acategory to a data item, store records of the category assignments towithin an index in memory, as well as generate revised visualrepresentations reflecting the categorization. Additional details ofmodules 122 and 124 are discussed in connection with FIGS. 3-14.

As shown in FIG. 1, in one embodiment, a network 140 is provided thatcan include various devices such as routers, server, and switchingelements connected in an Intranet, Extranet or Internet configuration.In one embodiment, the network 140 uses wired communications to transferinformation between access devices 150 and 160, the server device 110,the data store 130 and an administrative device 180. In anotherembodiment, the network 140 employs wireless communication protocols totransfer information between the access devices 150 and 160, the serverdevice 110, the data store 130 and the administrative device 180. Forexample, the network 140 may be a cellular or mobile network employingdigital cellular standards including but not limited to the 3GPP, 3GPP2and AMPS family of standards such as Global System for MobileCommunications (GSM), General Packet Radio Service (GPRS), CDMAOne,CDMA2000, Evolution-Data Optimized (EV-DO), LIE Advanced, Enhanced DataRates for GSM Evolution (EDGE), Universal Mobile TelecommunicationsSystem (UMTS), Digital Enhanced Cordless Telecommunications (DECT),Digital AMPS (IS-136/TDMA), and Integrated Digital Enhanced Network(iDEN). The network 140 may also be a Wide Area Network (WAN), such asthe Internet, which employs one or more transmission protocols, e.g.TCP/IP. As another example, the network 140 may employ a combination ofdigital cellular standards and transmission protocols. In yet otherembodiments, the network 140 may employ a combination of wired andwireless technologies to transfer information between the access devices150 and 160, the server device 110, the data store 130 and theadministrative device 180.

The data store 130 is a repository that maintains and stores informationutilized by the before-mentioned modules 122 and 124. In one embodiment,the data store 130 is a relational database. In another embodiment, thedata store 130 is a directory server, such as a Lightweight DirectoryAccess Protocol (“LDAP”). In yet another embodiment, the data store 130is an area of non-volatile memory 120 of the server device 110.

In one embodiment, as shown in the FIG. 1 example, the data store 130includes a credentials data store 132, a tool data store 134 and a deckdata store 136. According to one embodiment, the credentials data storemaintains logon credentials for administrators and end users. In oneembodiment, the tool data store 134 maintains the evaluation tool, alsoreferred herein as the administration interface tool. The deck datastore 136 maintains the one or more sets of data items, also referred toas “decks” or “decks of cards,” that need to be reviewed and evaluated,as well as those decks that have been evaluated and categorized alongwith a corresponding index outlining the categorized data items withtheir associated categories, labels and color-coding. In one embodiment,the deck data store 136 maintains the categorized set of data items in astructured data store, such as a relational or hierarchal database.

Although the data store 130 shown in FIG. 1 is connected to the network140, it will be appreciated by one skilled in the art that the datastore 130 and/or any of the information shown therein, can bedistributed across various servers and be accessible to the server 110over the network 140; be coupled directly to the server 110; beconfigured as part of server 110 and interconnected to processor 112,RAM 114, the one or more input-output devices 116 and the non-volatilememory 120 via the common bus; or be configured in an area ofnon-volatile memory 120 of the server 110.

The access devices 150 and 160, according to one embodiment, are generalpurpose or special purpose computing devices comprising: atouch-sensitive graphical user interface (“GUI”), GUI 154 and GUI 164,respectively; a digital signal processor (“DSP”), DSP 152 and DSP 162,respectively; each DSP having an access application module that allows auser to access the server 110, access application module 152A and accessapplication module 162A, respectively; transient and persistent storagedevices (not shown); an input/output subsystem (not shown); and a bus toprovide a communications path between components comprising the generalpurpose or special purpose computer (not shown). According to oneembodiment, access application module 152A and access application module162A are web-based and use thin client applications (not shown), such asa web browser, which allows a user to access the server 110. Examples ofweb browsers are known in the art, and include well-known web browserssuch as such as MICROSOFT® INTERNET EXPLORER®, GOOGLE CHROME™, MOZILLAFIREFOX® AND APPLE® SAFARI®. According to another embodiment, accessdevices 150 and 160, are mobile electronic devices, each having GUI, aDSP having an access application module, internal and external storagecomponents; a power management system; an audio component; audioinput/output components; an image capture and process system; RFantenna; and a subscriber identification module (SIM) (not shown).Although system 100 is described generally herein as comprising twoseparate access devices, access devices 150 and 160, it should beappreciated that the present invention does not require at least twoseparate access devices, nor is it limited to solely two access devices.Indeed, system 100 can include a single access device, such as accessdevice 150 or access device 160, or multiple access devices.

The administrator device 180, according to one embodiment, is a generalpurpose or special purpose computing device comprising a touch-sensitiveGUI 184, a DSP 182 having an administrator application module 182A thatallows a user to access the server 110, transient and persistent storagedevices (not shown), an input/output subsystem (not shown) and a bus toprovide a communications path between components comprising the generalpurpose or special purpose computer (not shown). According to oneembodiment, administrator application module 182A is web-based and usesa thin client application (not shown), such as a web browser, whichallows a user to access the server 110. Examples of web browsers areknown in the art, and include well-known web browsers such as such asMICROSOFT® INTERNET EXPLORER®, GOOGLE CHROME™, MOZILLA FIREFOX ANDAPPLE® SAFARI®. According to another embodiment, administrator device180 is a mobile device having a GUI (not shown), a digital signalprocessor with an application module (not shown), internal and externalstorage components (not shown), a power management system (not shown),an audio component (not shown), audio input/output components (notshown), an image capture and process system (not shown), RF antenna (notshown), and a subscriber identification module (SIM) (not shown).

Further, it should be noted that the system 100 shown in FIG. 1 is onlyone embodiment of the disclosure. Other system embodiments of thedisclosure may include additional structures that are not shown, such assecondary storage and additional computational devices. In addition,various other embodiments of the disclosure include fewer structuresthan those shown in FIG. 1. For example, in one embodiment, thedisclosure is implemented on a single computing device in anon-networked standalone configuration. Data input and requests arecommunicated to the computing device via an input device, such as akeyboard and/or mouse. Data output of the system is communicated fromthe computing device to a display device, such as a computer monitor.According to one embodiment, data output includes one or more ofcomputed significance scores, classifications assigned to the data setsunder review, identification of the reviewers who assigned theclassification, confidence scores of the reviewers, average scores ofthe assignments made by a set of reviewers, and scores based on thedegrees of agreement between the human and application classifications.

Turning now to FIG. 2, an example of a suitable computing system 200within which embodiments of the disclosure may be implemented ispresented. Specifically, FIG. 2 illustrates another embodiment of thepresent inventive system that utilizes a three tier hierarchal structureinvolving a super-administrator, administrators and reviewers ascompared to the system illustrated in FIG. 1 that utilizes a two tiersystem of administrators and reviewers. It should be noted that thesuper-administrator and administrators of system 200 serve in the samerole as the administrator of system 100 with the caveat that thesuper-administrator and administrators may each have discrete portionsof the overall roles served by the administrator of system 100. Further,the computing system 200 is another exemplary embodiment and is notintended to suggest any limitation as to the scope of use orfunctionality of the disclosure. Neither should the computing system 200be interpreted as having any dependency or requirement relating to anyone or combination of illustrated components. For example, the presentdisclosure is operational with numerous other general purpose or specialpurpose computing consumer electronics, network PCs, minicomputers,mainframe computers, laptop computers, as well as distributed computingenvironments that include any of the above systems or devices, and thelike.

According to one embodiment, the system 200 includes a server device 210configured to include a processor 212, such as a central processing unit(“CPU”), random access memory (“RAM”) 214, one or more input-outputdevices 216, such as a display device (not shown) and keyboard (notshown), non-volatile memory 220 and data store 225, all of which areinterconnected via a common bus and controlled by the processor 212.Additional details of non-volatile memory 220 and data store 225 arediscussed in connection with FIGS. 2-14.

As shown in FIG. 2, in one embodiment, a network 230 is provided thatcan include various devices such as routers, servers, and switchingelements connected in an Intranet, Extranet or Internet configuration.In one embodiment, the network 230 uses wired communications to transferinformation between access devices 250, 255 and 260, the server device210, the data store 225, administrator devices 280 and 281 and asuper-administrator device 245. In another embodiment, the network 230employs wireless communication protocols to transfer information betweenthe access devices 250, 255 and 260, the server device 210, the datastore 225, the administrator devices 280 and 281 and thesuper-administrator device 245. For example, the network 230 may be acellular or mobile network employing digital cellular standardsincluding but not limited to the 3GPP, 3GPP2 and AMPS family ofstandards such as Global System for Mobile Communications (GSM), GeneralPacket Radio Service (GPRS), CDMAOne, CDMA2000, Evolution-Data Optimized(EV-DO), LTE Advanced, Enhanced Data Rates for GSM Evolution (EDGE),Universal Mobile Telecommunications System (UMTS), Digital EnhancedCordless Telecommunications (DECT), Digital AMPS (IS-136/TDMA), andIntegrated Digital Enhanced Network (iDEN). The network 230 may also bea Wide Area Network (WAN), such as the Internet, which employs one ormore transmission protocols, e.g. TCP/IP. As another example, thenetwork 230 may employ a combination of digital cellular standards andtransmission protocols. In yet other embodiments, the network 230 mayemploy a combination of wired and wireless technologies to transferinformation between the access devices 250, 255 and 260, the serverdevice 210, the data store 225, the administrator devices 280 and 281and the super-administrator device 245.

The data store 225 is a repository that maintains and stores informationutilized by the before-mentioned modules 122 and 124. In one embodiment,the data store 225 is a relational database. In another embodiment, thedata store 225 is a directory server, such as a Lightweight DirectoryAccess Protocol (“LDAP”). In yet another embodiment, the data store 225is an area of non-volatile memory 220 of the server device 210.

Although the data store 225 shown in FIG. 2 is connected to the network230, it will be appreciated by one skilled in the art that the datastore 225 and/or any of the information shown therein, can bedistributed across various servers and be accessible to the server 210over the network 230; be coupled directly to the server 210; beconfigured as part of server 210 and interconnected to processor 212,RAM 214, the one or more input-output devices 216 and the non-volatilememory 220 via the common bus; or be configured in an area ofnon-volatile memory 220 of the server 210.

The access devices 250, 255 and 260, according to one embodiment, aregeneral purpose or special purpose computing devices comprising: atouch-sensitive graphical user interface (“GUI”), GUI 254, GUI 259 andGUI 264, respectively; a digital signal processor (“DSP”), DSP 252, DSP257 and DSP 262, respectively; each DSP having an access applicationmodule that allows a user to access the server 210, access applicationmodule 252A, access application module 257A and access applicationmodule 262A, respectively; transient and persistent storage devices (notshown); an input/output subsystem (not shown); and a bus to provide acommunications path between components comprising the general purpose orspecial purpose computer (not shown). According to one embodiment,access application module 252A, access application module 257A andaccess application module 262A are web-based and use thin clientapplications (not shown), such as a web browser, which allows a user toaccess the server 210. Examples of web browsers are known in the art,and include well-known web browsers such as such as MICROSOFT® INTERNETEXPLORER®, GOOGLE CHROME™, MOZILLA FIREFOX® AND APPLE® SAFARI®.According to another embodiment, access devices 250, 255 and 260, aremobile electronic devices, each having GUI, a DSP having an accessapplication module, internal and external storage components; a powermanagement system; an audio component; audio input/output components; animage capture and process system; RF antenna; and a subscriberidentification module (SIM) (not shown). Although system 200 isdescribed generally herein as comprising three separate access devices,access devices 250, 255 and 260, it should be appreciated that thepresent invention does not require at least three separate accessdevices, nor is it limited to solely three access devices. Indeed,system 200 can include a single access device, such as access device250, access device 255 or access device 260, or multiple access devices.

The administrator devices 280 and 281, according to one embodiment, aregeneral purpose or special purpose computing devices each comprising atouch-sensitive graphical user interface, GUI 284 and GUI 285,respectively; a digital signal processor, DSP 282 and DSP 284,respectively; each DSP having an access application module that allows auser to access the server 210, administrator application module 282A andadministrator application module 287A, respectively; transient andpersistent storage devices (not shown), an input/output subsystem (notshown) and a bus to provide a communications path between componentscomprising the general purpose or special purpose computer (not shown).According to one embodiment, administrator application modules 282A and287A are web-based and use a thin client application (not shown), suchas a web browser, which allows a user to access the server 210. Examplesof web browsers are known in the art, and include well-known webbrowsers such as such as MICROSOFT® INTERNET EXPLORER®, GOOGLE CHROME™,MOZILLA FIREFOX® and APPLE® SAFARI®. According to another embodiment,administrator devices 280 and 281 are mobile devices each having a GUI(not shown), a digital signal processor with an application module (notshown), internal and external storage components (not shown), a powermanagement system (not shown), an audio component (not shown), audioinput/output components (not shown), an image capture and process system(not shown), RF antenna (not shown), and a subscriber identificationmodule (SIM) (not shown).

The super-administrator device 245, according to one embodiment, is ageneral purpose or special purpose computing device comprising atouch-sensitive graphical user interface 247, a DSP 249 having asuper-administrator application module 249A that allows a user to accessthe server 210, transient and persistent storage devices (not shown), aninput/output subsystem (not shown) and a bus to provide a communicationspath between components comprising the general purpose or specialpurpose computer (not shown). According to one embodiment, thesuper-administrator application module 249A is web-based and uses a thinclient application (not shown), such as a web browser, which allows auser to access the server 210. Examples of web browsers are known in theart, and include well-known web browsers such as such as Microsoft®Internet Explorer®, Google Chrome™, Mozilla Firefox® and Apple® Safari®.According to another embodiment, the super-administrator device 245 is amobile device having a GUI (not shown), a digital signal processor withan application module (not shown), internal and external storagecomponents (not shown), a power management system (not shown), an audiocomponent (not shown), audio input/output components (not shown), animage capture and process system (not shown), RF antenna (not shown),and a subscriber identification module (SIM) (not shown).

Further, it should be noted that the system 200 shown in FIG. 2 is onlyone embodiment of the disclosure. Other system embodiments of thedisclosure may include additional structures that are not shown, such assecondary storage and additional computational devices. In addition,various other embodiments of the disclosure include fewer structuresthan those shown in FIG. 2. For example, in one embodiment, thedisclosure is implemented on a single computing device in anon-networked standalone configuration. Data input and requests arecommunicated to the computing device via an input device, such as akeyboard and/or mouse. Data output of the system is communicated fromthe computing device to a display device, such as a computer monitor.According to one embodiment, data output includes one or more ofcomputed significance scores, classifications assigned to the text spansunder review, identification of the reviewers who assigned theclassification, confidence scores of the reviewers, average scores ofthe assignments made by a set of reviewers, and scores based on thedegrees of agreement between the human and application classifications.

Turning now to FIG. 3, an exemplary method 300 for generating andassigning parameters for the evaluation tool is disclosed in the contextof system 100 of FIG. 1. It should be noted that the exemplarymethodology of FIG. 3, as well as the methodology and details discussedin conjunction with FIGS, 4-13, can also incorporated within the contextof system 200, wherein the super-administrator and administrators ofsystem 200 serve in the same role as the administrator of system 100with the caveat that the super-administrator and administrators may eachhave discrete portions of the overall roles served by the administratorof system 100.

Referring back to the illustrated embodiment shown in FIG. 3, log oncredentials are received at graphical user interface of theadministrator device 180, step 310. In one embodiment of the inventiveconcept, an administrator is designated with certain access and editingrights, and receives a user name and password, which are stored in thecredentials database 132 that are accessible by the administrativemodule 122 to verify the administrator and allow access to the system100. The administrator then accesses the server 110 via a website portalon the GUI 184, by entering the username and password, which aresubsequently compared by the administrator module 122 to theadministrator information stored in the Credentials database 132. FIG. 4illustrates an exemplary website portal 400 in which the administrator'se-mail and password 410 are required.

If the username and password combination are valid, the administratormodule 122 directs the user to an administration interface toolmaintained in the tool data store 134, step 320. According to oneembodiment, the administration interface tool is a secure webpagedashboard maintained in the Tools database 134 and comprises a UIAdministration Interface, a Deck Administration Interface and a UserAdministration Interface, which is presented to the administrator on theGUI 184. FIGS. 5, 6, and 13 illustrate an exemplary instance of theadministration interface tool, wherein FIG. 5 illustrates an exemplaryUI Administration Interface 500, FIG. 6 illustrates an exemplary UserAdministration Interface 600 and FIG. 13 illustrates an exemplary DeckAdministration Interface 1300. Depending upon a user's access rights,which are defined by data stored in the Credentials database 132 andassociated with the user's login information, the system 100 will allowthe administrator to view and edit certain information maintained withincertain fields in the Credentials database 132 pertaining to users ofthe access devices 150 and 160, as well as certain informationmaintained in the Tools database 134 pertaining to the data items to bereviewed.

Returning to step 320 of FIG. 3, the UI Administration interface of theadministration interface tool is presented to the administrator on thegraphical user interface 184. The administrator uses the UIAdministration interface to assign a name, a label and screen locationfor a visual representation of each of the set of categories to beassigned to a data item, step 330. FIG. 5 illustrates an exemplary UIAdministration Interface 500, wherein the administrator, after selectingthe UI Administration tab 505, can input the appropriate informationinto a “Name” field 510 for a given category to be assigned to dataitems. For example, according to one embodiment, categories to beassigned to data items include “Low Conformity,” “Medium Conformity,”“High Conformity,” “To Do,” and “Unsure,” which will correlate as to areviewer's determination as to how responsive a given data item is togiven topic or query. Further, for each category, the administrator canassign a color 520, a label 530 and a screen location 540 for eachvisual representation of each of the set of categories to be assigned toa data item, which is stored in the Tools database 134. For example,referring to FIG. 5, the category “Low Conformity” is assigned a label“LowConform,” the color “Red” and the “Right Screen Location.” Thevisual representation of each category according the parametersestablished in FIG. 5 is illustrated in FIGS. 8-12.

At step 340 of FIG. 3, the administrator uses the User AdministrationInterface of the administration interface tool to populate relevantinformation for each user that is to be assigned a deck to be analyzed.According to one embodiment, a deck is a collection of data items thatare to be reviewed and assigned one or more categories. FIG. 6illustrates an exemplary User Administration Interface 600 in which theadministrator, after selecting the Users Administration tab 605, enterseach user's: name into a “Name” field 620, e-mail address into an “EmailAddress” field 630 and role into a “Role” field 650. The administratorinterface tool maintained in the tool data store 134 then assigns aunique identifier, in this case a numerical identifier, to each user,which is entered in the “ID” field 610. The administrator using theadministrator interface tool maintained in the tool data store 134 alsoassigns a password to be used by the respective user in order to accessthe system 100, which is entered in “Password” field 640. Anadministrator, according to one embodiment, enters an additional userand populates the aforementioned user information by activating abutton, illustrated in FIG. 6 as the “Add New” button 660. Each user'sname and associated e-mail address, password, unique identifier and roleare maintained in the Credentials database 132.

Returning to FIG. 3, the administrator uses the Deck AdministrationInterface in order to generate one or more decks to be evaluated andthen assigns to each of the one or more users a deck to be evaluated,step 350. FIG. 13 illustrates an exemplary Deck Administration Interface1300 in which the administrator, after selecting the Deck Administrationtab 1305, generates a deck by populating a “deckname” field 1310 with anassigned deck name, selecting the “deck,” i.e. a collection or set ofdata items to be analyzed, using the “Choose File” button 1320 andpopulating field 1330 with the name of the user (or users) the deck isto be assigned to and reviewed by. According to one embodiment, one ormore decks are stored and maintained in the deck data store 136. In oneembodiment, the one or more decks are generated by randomly groupingdata items, such as documents or document segments, to be analyzed intoidentically numbered sets. For example, a thousand documents that needto be analyzed are randomly grouped into ten sets of one hundreddocuments. In another embodiment, the one or more decks are generated byrandomly grouping documents or document segments to be analyzed intosets not to exceed a predetermined threshold value. For example, apredetermined threshold value for each set may one hundred documents andin looking to 940 documents that need to be analyzed, the documentsrandomly grouped into nine sets of one hundred documents and one set offorty documents. In another embodiment, the one or more decks aregenerated by grouping like documents according to similarcharacteristics, such as the same document type.

Turning now to FIG. 7, an exemplary method 700 for evaluating andclassifying documents and document segments is disclosed. In theillustrated embodiment shown in FIG. 7, the administrative module 122 ofthe server 110 receives a first signal associated with a set ofcategories, the set of categories being administrator defined, step 710.According to one embodiment, a designated administrator uses the UIAdministration interface 500, as illustrated in FIG. 5, and inputs aname, a label and screen location for a visual representation of each ofthe set of categories to be assigned to a data item via the userinterface 184 of the administrator device 180, which is received by theadministrative module 122 of the server 110. For example, according toone embodiment, categories to be assigned to data items include “LowConformity,” “Medium Conformity,” “High Conformity,” “To Do,” and“Unsure,” which will correlate as to a reviewer's determination as tohow responsive a given data item is to given topic or query. Further,for each category, the administrator can assign a label and a screenlocation for the visual representation of each of the set of categoriesto be assigned to a data item, which is stored in the Tools database134. For example, referring to FIG. 5, the category “Low Conformity” isassigned a label “LowConform,” the color “Red” and the “Right ScreenLocation.”

At step 720, the administrative module 122 of the server 110 receives asecond signal associated with a first set of one or more data items tobe analyzed. According to one embodiment, the administrator uses theDeck Administration Interface 1300, as illustrated in FIG. 13 andpresented via the user interface 184 of the administrator device 180, toinput a deck name, select and upload the collection of data items to beanalyzed, and assign one or more users to review and categorize thedeck, which is received by the administrative module 122 of the server110. For example, the Deck Administration Interface 1300 at theinstruction of the administrator creates a deck named“Tax_Matters_Deck2.txt,” which includes a collection of data items to bereviewed, and assigns two users, Tero and Patrick, to undertake thereview.

The visual representations of the set of categories are generated by theadministrative module 122 and then presented on graphical userinterfaces 154 and/or 164, step 730. FIG. 8 illustrates an exemplarygraphical user interface 800 with a visual representation of each of thecategories being displayed along the perimeter of the interface. Forexample, category “High Conformity” is represented by the visualrepresentation High Conformity 810 located on the left side of theinterface, which may be color coded as green. Similarly, the categories“Medium Conformity,” “Low Conformity,” “Unsure” and “To Do” arerepresented by the visual representations Medium Conformity 811, LowConformity 812, Unsure 813 and Todo 814 located along the remainder ofthe perimeter of the interface, which may be color coded as yellow, red,violet and white, respectively.

At step 740, the administrative module 122 presents each of the one ormore data items of the first set to be analyzed on a card or tileoverlay on the graphical user interfaces 154 and 164. According to oneembodiment, an end user upon providing the appropriate logon credentialsis presented with one or more decks or sets of data items the revieweris currently assigned, as illustrated by the interface 1400 in FIG. 14.FIG. 8 illustrates the exemplary graphical user interface 800 withvisual representations of each of the cards or data items to be analyzedfrom a selected deck presented as a card or tile overlay. For example,data items 820 through 825 each represent text segments that are to beanalyzed and categorized as “High Conformity,” “Medium Conformity,” “LowConformity,” “Unsure” and “To Do.”

At step 750, the evaluation module 124 via the user interface 154 ofaccess device 150 or user interface 164 of access device 160 receives afirst swipe identification signal, the first swipe identification signalidentifying a first category from the set of categories to apply to afirst data item from the set of one or more data items to be analyzed.According to one embodiment, user interface 154 and user interface 164are touch sensitive displays that can be controlled through simple ormulti-touch gestures as is known in the art. In another embodiment, userinterface 154 and user interface 164 are touch sensitive displays thatin addition to being controlled through simple or multi-touch gestures,is also configured to be controlled through one or more peripheraldevices as is known in the art. It should be understood that while thefollowing description is described with regard to received swipeidentification signals, according to one embodiment, the categorizationof data item within the deck can also be performed using a combinationof simple or multi-touch gestures through the touch sensitive displaysand peripheral device commands.

Referring to FIG. 8, in one embodiment, a swipe identification signal isgenerated by an end user initiating a finger swipe gesture on a specificdata item, such as data item 820, and then dragging it to theappropriate category tab of the category tabs 810-814 located along theperimeter of the screen. For example, an end user at access device 150will review the content of data item 820, which is a text span thatreads “The obligations of the Association and the LISB Bank to effectthe Merger shall be subject to” and determine that it highly conforms tothe end user's understanding of representative text spans associatedwith merger clauses that are included in this particular data set of thedeck, and therefore should be categorized as “High Conformity.”Accordingly, the reviewer undertakes a finger swipe gesture on data item820 and drags the data item 820 to the left toward the “High Conformity”category tab 810 on the user interface 154 of access device 150 or userinterface 164 of access device 160. The evaluation module 124 at theserver 110 then receives a swipe identification signal corresponding tothe user's finger swipe gesture of data item 820 to the “HighConformity” tab 810.

Similarly, at step 760, the evaluation module 124 via the user interface154 of access device 150 or user interface 164 of access device 160receives a second swipe identification signal, the second swipeidentification signal identifying a second category from the set ofcategories to apply to a second data item from the set of one or moredata items to be analyzed. For example, again referring to FIG. 8, anend user at access device 160 will review the content of data item 824,which is a document excerpt that reads “The obligations of F&M Trust andFCNB to effect the Merger shall be subject to (i) the approval of this”and determine that it conforms moderately to the end user'sunderstanding of representative text spans associated with mergerclauses that are included in this particular data set of the deck andtherefore should be categorized as “Medium Conformity.” Accordingly, thereviewer undertakes a finger swipe gesture on data item 824 and dragsthe data item 820 to the top toward the “Medium Conformity” category tab811 on the user interface 164 of access device 160. The evaluationmodule 124 at the server 110 then receives a swipe identification signalcorresponding to the user's finger swipe gesture of data item 824 to the“Medium Conformity” tab 811.

Returning to FIG. 7, based upon the first and second swipeidentification signals, the evaluation module 124 generates a firstrecord and a second record, the first record associating the first dataitem with the first category and the second record associating thesecond data item with the second category, step 770, and stores thefirst and second records in the deck data store 136, step 780. Accordingto one embodiment, the evaluation module 124 stores and maintains arecord of the data item and the corresponding selected category in anindex format within the deck data store 130. In another embodiment, theevaluation module 124 assigns and associates a label and/or color codingto the categorized data item record within the index. Continuing fromthe previous example and referring to FIG. 8, the evaluation module 124having received a first swipe identification signal that data item 820has been categorized as “High Conformity” and that data item 824 hasbeen categorized as “Medium Conformity,” generates a record of eachreceived signal and enters it into an index maintained in the deck datastore 130. In one embodiment, the record for data item 820 would includeidentification for the data item, the corresponding selected category“High Conformity,” and the corresponding label “HighConform.” Similarly,the record for data item 824 would include identification for the dataitem, the corresponding selected category “Medium Conformity,” and thecorresponding label “MediumConform.”

According to one embodiment, subsequent to the evaluation module 124receiving one or more swipe identification signals, the evaluationmodule 124 generates a modified visual representation of the data itemas illustrated in FIGS. 9 and 12. For example, referring to FIG. 9, dataitems 920 and 924 have been categorized by the end user via theaforementioned described in relation to FIG. 7 and are representative ofa revised visual representation having the associated category labelsaffixed to them. Specifically, for example, data item 920 of FIG. 9 isaffixed with the label “HighConform” and data item 924 is affixed withthe label “MediumConform.” Referring again to FIG. 9, the administrationinterface tool, in one embodiment, further allows an end user to modifycategory assignments to data items prior to completing her evaluation,which is signaled by the end user by pressing the button “Done” 915. Forexample, referring to FIG. 9, data item 925 has been categorized as“Unsure”; prior to the end user indicating that she is done by pressingbutton 915, the end user can alter the category assignment byundertaking a finger swipe gesture on data item 925 and dragging it tothe appropriate category tabs 910-912 located along the perimeter of thescreen.

In one embodiment, the evaluation tool is configured to allow an enduser to view all categorized data items that have been categorizedaccording to its assigned category. Referring to FIGS. 10 and 11, an enduser can view all data items that have been evaluated and assigned acertain category by performing a touch gesture on one of the categorytabs along the perimeter. For example, referring to FIG. 10, the enduser using her finger touches the “Medium Conformity” tab 1011 and theevaluation module 124 receives a touch identification signal that alldata items categorized as “Medium Conformity” are requested to bedisplayed, which prompts the evaluation module 124 to in turn toidentify all data items that have been categorized as “MediumConformity” within the index maintained in deck data store 136 and togenerate and display visual representations of each such data item, asillustrated in FIG. 10. Similarly in FIG. 11, the evaluation moduledisplays all data items that have been categorized as “To Do.” In thisfashion, an end user can quickly identify at any point of her evaluationwhat data items have been categorized, what category has been applied tothe evaluated data items, as well as having the ability to categorizesomething with a temporary category, such as “Unsure” or “To Do” andallow the end user to return at a later time to finally categorize thedata item. In another embodiment, temporary categories such as “Unsure”or “To do” are included in the end user's evaluation, which allows foran administrator or super administrator to further evaluate thosecategorized data items.

According to one embodiment, an administrator or super administrator isnotified of data items categorized as “Unsure,” for example, by reviewof a report generated from the administration interface tool. Referringto FIG. 13, in one embodiment, an administrator via the user interface184 of the administrator device 184 can elect to generate a summaryreport of the completed evaluation of data sets or decks by pressing thebutton 1344 of the deck administration interface illustrated in FIG. 13,which instructs the administrative module 122 to generate the summaryreport from records maintained in an index in deck data store 136.

Turning now to FIG. 15, a screen diagram of an exemplary summary reportoutlining exemplary classification statistics and results from theinterface tool is disclosed. In the embodiment shown in FIG. 15, anexemplary summary report 1500 is generated by the administrative module122 in response to an election by an administrator who selected button1344 of the deck administration interface illustrated in FIG. 13.According to one embodiment, the summary report 1500 is generated usinga spreadsheet application as is known in the art, such as MICROSOFT®EXCEL®. In one embodiment, the summary report 1500 includes one or moreclassifications of each evaluated data item, along with thecorresponding reviewer for each classification attributed to the dataitem. For example, referring to the summary report 1500, text segment1510 has been evaluated by Reviewer 1 Lora who categorized the segmentwith a“Low Conformity” label at 1522, which corresponds to a class scoreof 1 at 1524. According to one embodiment, a class score is set by theadministrator and corresponds to a given category. For example, “HighConformity” is associated with a class score of 3, “Medium Conformity”is associated with a class score of 2 and “Low Conformity” is associatedwith a class score of 1. Referring back to the summary report 1500, textsegment 1510 has also been evaluated by Reviewer 2 Tero, who alsocategorized the segment with a “Low Conformity” label at 1532, whichcorresponds to a class score of 1 at 1534.

According to one embodiment, the administrative module 122 is configuredto generate one or more statistical data points summarizing the one ormore classifications assigned to each evaluated document segment. Forexample, as illustrated in FIG. 15, a classification average score 1540,a classification variance score 1542, a classification standarddeviation score 1544 and a classification normalized average score 1546is determined based on the three classification assigned to documentsegment 1510. According to one embodiment, the statistical data pointscan be based on analysis of individual classification scores, combinedclassification scores, temporal attributes of individual evaluations andcombined evaluations and comparison among reviewers and with machineclassification determinations.

FIGS. 1 through 15 are conceptual illustrations allowing for anexplanation of the present disclosure. It should be understood thatvarious aspects of the embodiments of the present disclosure could beimplemented in hardware, firmware, software, or combinations thereof. Insuch embodiments, the various components and/or steps would beimplemented in hardware, firmware, and/or software to perform thefunctions of the present disclosure. That is, the same piece ofhardware, firmware, or module of software could perform one or more ofthe illustrated blocks (e.g., components or steps).

In software implementations, computer software (e.g., programs or otherinstructions) and/or data is stored on a machine readable medium as partof a computer program product, and is loaded into a computer system orother device or machine via a removable storage drive, hard drive, orcommunications interface. Computer programs (also called computercontrol logic or computer readable program code) are stored in a mainand/or secondary memory, and executed by one or more processors(controllers, or the like) to cause the one or more processors toperform the functions of the disclosure as described herein. In thisdocument, the terms “machine readable medium,” “computer program medium”and “computer usable medium” are used to generally refer to media suchas a random access memory (RAM); a read only memory (ROM); a removablestorage unit (e.g., a magnetic or optical disc, flash memory device, orthe like); a hard disk; or the like.

Notably, the figures and examples above are not meant to limit the scopeof the present disclosure to a single embodiment, as other embodimentsare possible by way of interchange of some or all of the described orillustrated elements. Moreover, where certain elements of the presentdisclosure can be partially or fully implemented using known components,only those portions of such known components that are necessary for anunderstanding of the present disclosure are described, and detaileddescriptions of other portions of such known components are omitted soas not to obscure the disclosure. In the present specification, anembodiment showing a singular component should not necessarily belimited to other embodiments including a plurality of the samecomponent, and vice-versa, unless explicitly stated otherwise herein.Moreover, the applicants do not intend for any term in the specificationor claims to be ascribed an uncommon or special meaning unlessexplicitly set forth as such. Further, the present disclosureencompasses present and future known equivalents to the known componentsreferred to herein by way of illustration.

The foregoing description of the specific embodiments so fully revealsthe general nature of the disclosure that others can, by applyingknowledge within the skill of the relevant art(s), readily modify and/oradapt for various applications such specific embodiments, without undueexperimentation, without departing from the general concept of thepresent disclosure. Such adaptations and modifications are thereforeintended to be within the meaning and range of equivalents of thedisclosed embodiments, based on the teaching and guidance presentedherein. It is to be understood that the phraseology or terminologyherein is for the purpose of description and not of limitation, suchthat the terminology or phraseology of the present specification is tobe interpreted by the skilled artisan in light of the teachings andguidance presented herein, in combination with the knowledge of oneskilled in the relevant art(s).

While various embodiments of the present disclosure have been describedabove, it should be understood that they have been presented by way ofexample, and not as limitations. It would be apparent to one skilled inthe relevant art(s) that various changes in form and detail could bemade therein without departing from the spirit and scope of thedisclosure. Thus, the present disclosure should not be limited by any ofthe above-described exemplary embodiments, but should be defined only inaccordance with the following claims and their equivalents.

What we claim is:
 1. A computer-implemented method for classifying datacomprising: receiving a first signal associated with a set ofcategories, the set of categories being user defined; receiving at aserver a second signal associated with a first set of one or more dataitems to be analyzed; presenting a selectable first category tab along afirst side of a perimeter of a touch-sensitive graphical user interface;presenting a selectable second category tab along a second side of theperimeter of the touch-sensitive graphical user interface, wherein thesecond category tab corresponds to a continuous region of the secondside of the perimeter of the touch-sensitive graphical user interface;presenting a selectable third category tab along a third side of theperimeter of the touch-sensitive graphical user interface, wherein thethird category tab corresponds to a continuous region of the third sideof the perimeter of the touch-sensitive graphical user interface;presenting a selectable fourth category tab along a fourth side of theperimeter of the touch-sensitive graphical user interface, wherein thefourth category tab corresponds to a continuous region of the fourthside of the perimeter of the touch-sensitive graphical user interface,the first, second, third, and fourth category tabs define a display areaon the of the touch-sensitive graphical user interface; presenting eachof the one or more data items of the first set to be analyzed as a tileon a tile overlay on the touch-sensitive graphical user interface,wherein each of the one or more data items comprise at least onedocument and wherein the tile associated with each of the one or moredata items comprises a visual representation of a segment of thedocument, and wherein the one or more data items are presented in thedisplay area in a grid with the first second third and fourth categorytabs surrounding the one or more data items; receiving a first swipeidentification signal, said first swipe identification signal beingdirectionally oriented toward the first category tab to identify a firstcategory from the set of categories to apply to a first data item fromthe first set of one or more data items to be analyzed, wherein thefirst category is a first level of conformity with respect to a topic orquery; receiving a second swipe identification signal, said second swipeidentification signal being directionally oriented toward the secondcategory tab to identify a second category from the set of categories toapply to a second data item from the first set of one or more data itemsto be analyzed, wherein the second category is a second level ofconformity with respect to the topic or query; receiving a third swipeidentification signal, said third swipe identification signal beingdirectionally oriented toward the third category tab to identify a thirdcategory from the set of categories to apply to a third data item fromthe first set of one or more data items to be analyzed, wherein thethird category is a third level of conformity with respect to the topicor query; receiving a fourth swipe identification signal, said fourthswipe identification signal being directionally oriented toward thefourth category tab to identify a fourth category from the set ofcategories to apply to a fourth data item from the first set of one ormore data items to be analyzed, wherein the fourth category is a fourthlevel of conformity with respect to the topic or query; wherein thefirst level of conformity is a higher degree of conformity as comparedto the second level of conformity, the second level of conformity is ahigher degree of conformity as compared to the third level of conformityand the fourth level of conformity is an uncertainty of conformitydecision with respect to the topic or query; wherein the first swipeidentification signal comprises initiating a first finger swipe gestureon the first data item and dragging the first data item to the firstcategory tab representative of the first category located along theperimeter of the touch-sensitive graphical user interface, and whereinin response to the first swipe identification signal affixing a firstcategory label to the tile associated with the first data item in thetouch-sensitive graphical user interface; wherein the second swipeidentification signal comprises initiating a second finger swipe gestureon the second data item and dragging the second data item to the secondcategory tab representative of the second category located along theperimeter of the touch-sensitive graphical user interface, and whereinin response to the second swipe identification signal affixing a secondcategory label to the tile associated with the second data item in thetouch-sensitive graphical user interface; wherein the third swipeidentification signal comprises initiating a third finger swipe gestureon the third data item and dragging the third data item to the thirdcategory tab representative of the third category located along theperimeter of the touch-sensitive graphical user interface, and whereinin response to the third swipe identification signal affixing a thirdcategory label to the tile associated with the third data item in thetouch-sensitive graphical user interface; wherein the fourth swipeidentification signal comprises initiating a fourth finger swipe gestureon the fourth data item and dragging the fourth data item to the fourthcategory tab representative of the fourth category located along theperimeter of the touch-sensitive graphical user interface, and whereinin response to the fourth swipe identification signal affixing a fourthcategory label to the tile associated with the fourth data item in thetouch-sensitive graphical user interface; in response to a selection ofat least one of the first, second, third, and fourth category tabs,identifying from the first set of data items, data items associated withthe selected at least one of the first, second, third, and fourthcategory tabs and updating the touch-sensitive graphical user interfaceto display in the display area only the data items associated with theselected at least one of the first, second, third, and fourth categorytabs: based upon the first, second, third, and fourth swipeidentification signals, generating a first record, a second record, athird record and a fourth record, the first record associating the firstdata item with the first category, the second record associating thesecond data item with the second category, the third record associatingthe third data item with the third category and the fourth recordassociating the fourth data item with the fourth category; storing thefirst, second, third, and fourth records in a memory; and generating asummary report comprising a set of statistical data values to each ofthe first, second, third, and fourth records, a given set of statisticaldata values corresponding to one more categories assigned to a givendata item, said given set of statistical values comprising aclassification average score, a classification variance score, aclassification standard deviation score and a classification normalizedaverage score.
 2. The computer-implemented method of claim 1 furthercomprising generating a spreadsheet file comprising the first, secondand third records.
 3. The computer-implemented method of claim 1 furthercomprising displaying an administration interface tool comprising a UIadministration interface, a deck administration interface and a useradministration interface.
 4. The computer-implemented method of claim 3further comprising populating a category name field, a category labelfield and a screen location field in the UI administration interface foreach category of the set of categories to be assigned a data item, thescreen location field populated with a screen location valuerepresentative of the location on the graphical user interface of avisual representation of each category.
 5. The computer-implementedmethod of claim 4 further comprising: populating an end user name fieldand an end user email address field for one or more end users in theuser administration interface; and assigning an end user identification,an end user password and an end user role to each of the one or more endusers in the user administration interface.
 6. The computer-implementedmethod of claim 4 further comprising: generating, using the deckadministration interface, one or more sets of one or more data items tobe analyzed; and assigning, using the deck administration interface, oneor more end users to the one or more sets of one or more data items tobe analyzed.
 7. The method of claim 1, wherein the first category tabcorresponds to another continuous region encompassing an entirety of atotal length of the first side of the perimeter of the touch-sensitivegraphical user interface, and the first side of the perimeter of thetouch-sensitive graphical user interface is adjacent to the second sideof the perimeter of the touch-sensitive graphical user interface, andwherein the second side of the perimeter of the touch-sensitivegraphical user interface is adjacent to the third side of the perimeterof the touch-sensitive graphical user interface, therewith presentingthe first, second, and third category tabs in order according to degreeof conformity with respect to the topic or query.
 8. The method of claim1, wherein a total number of category tabs is greater than another totalnumber of sides of the perimeter of the touch-sensitive graphical userinterface, and an entirety of at least one side of the perimeter of thetouch-sensitive graphical user interface is evenly divided into two ormore continuous regions each corresponding to a category tab. 9.Non-transitory computer readable media comprising program code storedthereon for execution by a programmable processor to perform a methodfor classifying data, the non-transitory computer readable mediacomprising: program code for receiving a first signal associated with aset of categories, the set of categories being user defined; programcode for receiving a second signal associated with a first set of one ormore data items to be analyzed; program code for presenting a selectablefirst category tab along a first side of a perimeter of a graphical userinterface; program code for presenting a selectable second category tabalong a second side of the perimeter of the graphical user interface,wherein the second category tab corresponds to a continuous region ofthe second side of the perimeter of the graphical user interface;program code for presenting a selectable third category tab along athird side of the perimeter of the touch-sensitive graphical userinterface, wherein the third category tab corresponds to a continuousregion of the third side of the perimeter of the touch-sensitivegraphical user interface; program code for presenting a selectablefourth category tab along a fourth side of the perimeter of thetouch-sensitive graphical user interface, wherein the fourth categorytab corresponds to a continuous region of the fourth side of theperimeter of the touch-sensitive graphical user interface. the first,second, third, and fourth category tabs define a display area on the ofthe touch-sensitive graphical user interface; program code forpresenting each of the one or more data items of the first set to beanalyzed on a tile overlay on the graphical user interface. wherein eachof the one or more data items comprise at least one document and whereinthe tile associated with each of the one or more data items comprises avisual representation of a segment of the document, and wherein the oneor more data items are presented in the display area in a grid with thefirst, second, third, and fourth category tabs surrounding the one ormore data items; program code for receiving a first swipe identificationsignal, said first swipe identification signal being directionallyoriented toward the first category tab to identify a first category fromthe set of categories to apply to a first data item from the first setof one or more data items to be analyzed, wherein the first category isa first level of conformity with respect to the topic or query; programcode for receiving a second swipe identification signal, said secondswipe identification signal being directionally oriented toward thesecond category tab to identify a second category from the set ofcategories to apply to a second data item from the first set of one ormore data items to be analyzed, wherein the second category is a secondlevel of conformity with respect to the topic or query; program code forreceiving a third swipe identification signal, said third swipeidentification signal being directionally oriented toward the thirdcategory tab to identify a third category from the set of categories toapply to a third data item from the first set of one or more data itemsto be analyzed, wherein the third category is a third level ofconformity with respect to the topic or query; program code forreceiving a fourth swipe identification signal, said fourth swipeidentification signal being directionally oriented toward the fourthcategory tab to identify a fourth category from the set of categories toapply to a fourth data item from the first set of one or more data itemsto be analyzed, wherein the fourth category is a fourth level ofconformity with respect to the topic or query; wherein the first levelof conformity is a higher degree of conformity as compared to the secondlevel of conformity, the second level of conformity is a higher degreeof conformity as compared to the third level of conformity and thefourth level of conformity is an uncertainty of conformity decision withrespect to the topic or query; wherein the first swipe identificationsignal comprises initiating a first finger swipe gesture on the firstdata item and dragging the first data item to the first category tabrepresentative of the first category located along the perimeter of thegraphical user interface, and wherein in response to the first swipeidentification signal affixing a first category label to the tileassociated with the first data item in the touch-sensitive graphicaluser interface; wherein the second swipe identification signal comprisesinitiating a second finger swipe gesture on the second data item anddragging the second data item to the second category tab representativeof the second category located along the perimeter of the graphical userinterface, and wherein in response to the second swipe identificationsignal affixing a second category label to the tile associated with thesecond data item in the touch-sensitive graphical user interface;wherein the third swipe identification signal comprises initiating athird finger swipe gesture on the third data item and dragging the thirddata item to the third category tab representative of the third categorylocated along the perimeter of the touch-sensitive graphical userinterface, and wherein in response to the third swipe identificationsignal affixing a third category label to the tile associated with thethird data item in the touch-sensitive graphical user interface; whereinthe fourth swipe identification signal comprises initiating a fourthfinger swipe gesture on the fourth data item and dragging the fourthdata item to the fourth category tab representative of the fourthcategory located along the perimeter of the touch-sensitive graphicaluser interface, and wherein in response to the fourth swipeidentification signal affixing a fourth category label to the tileassociated with the fourth data item in the touch-sensitive graphicaluser interface; program code for, in response to a selection of at leastone of the first, second, third, and fourth category tabs, identifyingfrom the first set of data items, data items associated with theselected at least one of the first, second, third, and fourth categorytabs and updating the touch-sensitive graphical user interface todisplay in the display area only the data items associated with theselected at least one of the first, second, third, and fourth categorytabs; program code for, based upon the first, second, third, and fourthswipe identification signals, generating a first record, a secondrecord, a third record and a fourth record, the first record associatingthe first data item with the first category, the second recordassociating the second data item with the second category, the thirdrecord associating the third data item with the third category and thefourth record associating the fourth data item with the fourth category;program code for storing the first, second, third, and fourth records ina memory; and program code for generating a summary report comprising aset of statistical data values to each of the first, second, third, andfourth records, a given set of statistical data values corresponding toone more categories assigned to a given data item, said given set ofstatistical values comprising a classification average score, aclassification variance score, a classification standard deviation scoreand a classification normalized average score.
 10. The non-transitorycomputer readable media of claim 9 further comprising program code forgenerating a spreadsheet file comprising the first, second and thirdrecords.
 11. The non-transitory computer readable media of claim 9further comprising program code for displaying an administrationinterface tool comprising a UI administration interface, a deckadministration interface and a user administration interface.
 12. Thenon-transitory computer readable media of claim 11 further comprisingprogram code for populating a category name field, a category labelfield and a screen location field in the UI administration interface foreach category of the set of categories to be assigned a data item, thescreen location field populated with a screen location valuerepresentative of the location on the graphical user interface of avisual representation of each category.
 13. The non-transitory computerreadable media of claim 12 further comprising: program code forpopulating an end user name field and an end user email address fieldfor one or more end users in the user administration interface; andprogram code for assigning an end user identification, an end userpassword and an end user role to each of the one or more end users inthe user administration interface.
 14. The non-transitory computerreadable media of claim 12 further comprising: program code forgenerating, using the deck administration interface, one or more sets ofone or more data items to be analyzed; and program code for assigning,using the deck administration interface, one or more end users to theone or more sets of one or more data items to be analyzed.
 15. A systemfor classifying data, the system comprising: a data repositorycomprising a first memory; and a server including a processor configuredto: receive a first signal associated with a set of categories, the setof categories being user defined; receive a second signal associatedwith a first set of one or more data items to be analyzed; present aselectable first category tab along a first side of a perimeter of atouch-sensitive graphical user interface; present a selectable secondcategory tab along a second side of the perimeter of the touch-sensitivegraphical user interface, wherein the second category tab corresponds toa continuous region of the second side of the perimeter of thetouch-sensitive graphical user interface; present a selectable thirdcategory tab along a third side of the perimeter of the touch-sensitivegraphical user interface, wherein the third category tab corresponds toa continuous region of the third side of the perimeter of thetouch-sensitive graphical user interface; present a selectable fourthcategory tab along a fourth side of the perimeter of the touch-sensitivegraphical user interface, wherein the fourth category tab corresponds toa continuous region of the fourth side of the perimeter of thetouch-sensitive graphical user interface, the first, second, third, andfourth category tabs define a display area on the of the touch-sensitivegraphical user interface; present each of the one or more data items ofthe first set to be analyzed on a tile overlay on the touch-sensitivegraphical user interface, wherein each of the one or more data itemscomprise at least one document and wherein the tile associated with eachof the one or more data items comprises a visual representation of asegment of the document, and wherein the one or more data items arepresented in the display area in a grid with the first, second, third,and fourth category tabs surrounding the one or more data items; receivea first swipe identification signal, said first swipe identificationsignal being directionally oriented toward the first category tab toidentify a first category from the set of categories to apply to a firstdata item from the first set of one or more data items to be analyzed,wherein the first category is a first level of conformity with respectto a topic or query; receive a second swipe identification signal, saidsecond swipe identification signal being directionally oriented towardthe second category tab to identify a second category from the set ofcategories to apply to a second data item from the first set of one ormore data items to be analyzed, wherein the second category is a secondlevel of conformity with respect to the topic or query; receive a thirdswipe identification signal, said third swipe identification signalbeing directionally oriented toward the third category tab to identify athird category from the set of categories to apply to a third data itemfrom the first set of one or more data items to be analyzed, wherein thethird category is a third level of conformity with respect to the topicor query; receive a fourth swipe identification signal, said fourthswipe identification signal being directionally oriented toward thefourth category tab to identify a fourth category from the set ofcategories to apply to a fourth data item from the first set of one ormore data items to be analyzed, wherein the fourth category is a fourthlevel of conformity with respect to the topic or query; wherein thefirst level of conformity is a higher degree of conformity as comparedto the second level of conformity, the second level of conformity is ahigher degree of conformity as compared to the third level of conformityand the fourth level of conformity is an uncertainty of conformitydecision with respect to the topic or query; wherein the first swipeidentification signal comprises initiating a first finger swipe gestureon the first data item and dragging the first data item to the firstcategory tab representative of the first category located along theperimeter of the touch-sensitive graphical user interface, and whereinin response to the first swipe identification signal affixing a firstcategory label to the tile associated with the first data item in thetouch-sensitive graphical user interface; wherein the second swipeidentification signal comprises initiating a second finger swipe gestureon the second data item and dragging the second data item to the secondcategory tab representative of the second category located along theperimeter of the touch-sensitive graphical user interface, and whereinin response to the second swipe identification signal affixing a secondcategory label to the tile associated with the second data item in thetouch-sensitive graphical user interface; wherein the third swipeidentification signal comprises initiating a third finger swipe gestureon the third data item and dragging the third data item to the thirdcategory tab representative of the third category located along theperimeter of the touch-sensitive graphical user interface, and whereinin response to the third swipe identification signal affixing a thirdcategory label to the tile associated with the third data item in thetouch-sensitive graphical user interface; wherein the fourth swipeidentification signal comprises initiating a fourth finger swipe gestureon the fourth data item and dragging the fourth data item to the fourthcategory tab representative of the fourth category located along theperimeter of the touch-sensitive graphical user interface, and whereinin response to the fourth swipe identification signal affixing a fourthcategory label to the tile associated with the fourth data item in thetouch-sensitive graphical user interface; in response to a selection ofat least one of the first, second, third, and fourth category tabs,identifying from the first set of data items, data items associated withthe selected at least one of the first, second, third, and fourthcategory tabs and updating the touch-sensitive graphical user interfaceto display in the display area only the data items associated with theselected at least one of the first, second, third, and fourth categorytabs based upon the first, second, third, and fourth swipeidentification signals, generate a first record, a second record, athird record and a fourth record, the first record associating the firstdata item with the first category, the second record associating thesecond data item with the second category, the third record associatingthe third data item with the third category and the fourth recordassociating the fourth data item with the fourth category; store thefirst, second, third, and fourth records in a memory; and generate asummary report comprising a set of statistical data values to each ofthe first, second, third, and fourth records, a given set of statisticaldata values corresponding to one more categories assigned to a givendata item, said given set of statistical values comprising aclassification average score, a classification variance score, aclassification standard deviation score and a classification normalizedaverage score.
 16. The system of claim 15 wherein the server is furtherconfigured to generate a spreadsheet file comprising the first, secondand third records.
 17. The system of claim 15 wherein the server isfurther configured to display an administration interface toolcomprising a UI administration interface, a deck administrationinterface and a user administration interface.
 18. The system of claim15 wherein the server is further configured to populate a category namefield, a category label field and a screen location field in a UIadministration interface for each category of the set of categories tobe assigned a data item, the screen location field populated with ascreen location value representative of a location on the graphical userinterface of a visual representation of each category.
 19. The system ofclaim 18 wherein the server is further configured to: populate an enduser name field and an end user email address field for one or more endusers in a user administration interface; and assign an end useridentification, an end user password and an end user role to each of theone or more end users in the user administration interface.
 20. Thesystem of claim 18 wherein the server is further configured to:generate, using a deck administration interface, one or more sets of oneor more data items to be analyzed; and assign, using the deckadministration interface, one or more end users to the one or more setsof one or more data items to be analyzed.